Repository navigation
[AINode] Add Timestral as a built-in forecasting model, downloaded on load - #19272
kotselidis wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Malformed metadata can still trigger inference-pool creation, combined-series attention can exhaust memory, and slow downloads block management operations and delay shutdown.
3 open issues
What changed in this PR
Adds Timestral-209M as an AINode built-in forecasting model. Its restricted-license weights are downloaded explicitly through LOAD MODEL, rather than automatically at startup.
Changes:
- Adds Timestral inference code and a median-forecast pipeline supporting multivariate inputs, covariates, and multi-block horizons.
- Introduces on-request downloads, model-state handling, and RPC checks for unavailable weights.
- Adds unit and integration coverage and updates third-party license notices.
| File | Description |
|---|---|
NOTICE |
Adds Timestral attribution. |
LICENSE |
Records the vendored code’s Apache license. |
iotdb-core/ainode/tests/test_timestral.py |
Tests checkpoint loading, forecasts, and covariates. |
iotdb-core/ainode/tests/test_builtin_model_download.py |
Tests download states, retries, and pre-download rejection. |
iotdb-core/ainode/src/assembly/resources/NOTICE-binary |
Adds attribution to AINode binary notices. |
iotdb-core/ainode/iotdb/ainode/core/rpc/handler.py |
Rejects inference before on-request weights are available. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/scaler.py |
Implements normalization and forecast rescaling. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/pipeline_timestral.py |
Adapts AINode inputs and returns median forecasts. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/modeling_timestral.py |
Integrates strict checkpoint loading and device transitions. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/model.py |
Implements the Timestral network and quantile prediction. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/masks.py |
Defines attention masks and temporal positions. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/layers.py |
Adds transformer layers and rotary embeddings. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/forecaster.py |
Handles block decoding, covariates, and forecast adjustments. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/covariate_encoding.py |
Encodes categorical covariates numerically. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/config.py |
Defines and validates checkpoint configuration. |
iotdb-core/ainode/iotdb/ainode/core/model/timestral/__init__.py |
Adds the package initializer and attribution. |
iotdb-core/ainode/iotdb/ainode/core/model/model_storage.py |
Manages deferred downloads and stored-model states. |
iotdb-core/ainode/iotdb/ainode/core/model/model_info.py |
Registers Timestral and adds the startup-download flag. |
iotdb-core/ainode/iotdb/ainode/core/manager/model_manager.py |
Exposes built-in model downloads to callers. |
iotdb-core/ainode/iotdb/ainode/core/inference/pool_controller.py |
Downloads deferred weights before loading inference pools. |
integration-test/src/test/java/org/apache/iotdb/ainode/utils/AINodeTestUtils.java |
Defines expected on-request built-in models. |
integration-test/src/test/java/org/apache/iotdb/ainode/it/AINodeSharedClusterIT.java |
Covers model listing and load/inference/forecast flow. |
Review effort: Balanced
| succeeded = self._download_builtin_model_files(model_id, model_dir) | ||
| finally: | ||
| self._set_model_download_result(model_id, succeeded) | ||
| return succeeded |
There was a problem hiding this comment.
Reproduced with a malformed local config.json: download_builtin_model("timestral") returns True even though _set_model_download_result() leaves the model INACTIVE. The controller consequently proceeds to create a pool whose checkpoint cannot load. Please propagate configuration-processing failure into the returned result and add a regression test for malformed JSON.
There was a problem hiding this comment.
Fixed in fb8f3b4: when the downloaded config.json cannot be read, the model is left INACTIVE and the download now reports failure, so LOAD MODEL fails instead of creating a pool. test_builtin_model_download.py covers a malformed config.json.
| Forecast the series, batching those of the same shape without covariates. | ||
|
|
||
| Returns | ||
| ------- | ||
| list of torch.Tensor | ||
| One tensor per series, of shape (target_count, output_length, quantile_count). | ||
| """ | ||
| output_length = infer_kwargs.get("output_length", 96) | ||
| outputs: list[torch.Tensor | None] = [None] * len(inputs) | ||
| batches: dict[tuple, list[int]] = {} | ||
| for i, item in enumerate(inputs): | ||
| if item["past_covariates"] is None and item["known_covariates"] is None: | ||
| batches.setdefault(tuple(item["context"].shape), []).append(i) | ||
| else: | ||
| batches[("series", i)] = [i] |
There was a problem hiding this comment.
I checked the actual mask builder with meta tensors: one 100-target series with 1,056 patch slots produces a dense boolean mask of shape [1056, 1, 100, 100] (10,560,000 bytes), while ten such series produce [1056, 1, 1000, 1000] (1,056,000,000 bytes). That is 100 times the mask memory for ten times the input; the normal request scheduler can select ten requests at once. Please preserve a separate batch dimension for independent series or process them separately/in bounded batches, and add a regression covering multiple concurrent requests.
There was a problem hiding this comment.
Fixed in 8e171ce: series of the same shape are batched only up to 64 target rows (max_batch_rows), and a new batch starts beyond that, so the variate-attention mask of a batch stays bounded and memory grows linearly with the number of requests. test_timestral.py checks the split.
| model_info = self._model_manager.get_model_info(model_id) | ||
| if not model_info.download_on_start: | ||
| # Built-in models that are not downloaded at start are downloaded when loaded | ||
| if not self._model_manager.download_builtin_model(model_id): |
There was a problem hiding this comment.
Confirmed with an event-controlled download: a task queued behind LOAD MODEL timestral does not execute until the download completes. This is the controller's only management worker, and stop() joins it before closing existing pools, so a slow download also delays unrelated unloads and shutdown. Please move the transfer off this queue and make shutdown cancellation bounded, while preserving ordering for the affected model.
There was a problem hiding this comment.
The download now leaves unrelated controller operations and shutdown unblocked. One ordering gap remains through the public API: InferenceManager.unload_model() only forwards devices with an existing request pool. While Timestral is downloading, no pool exists, so UNLOAD MODEL timestral FROM DEVICES 'cpu' returns success without reaching _tasks_waiting_for_download; the model is then loaded when the download completes.
I reproduced this using the event-controlled download and the real manager methods: the unload returned status 200, there were zero deferred tasks, and completion produced only a load event. Please make pending loads visible to unload handling, or otherwise forward the unload for ordering, and cover LOAD → UNLOAD through InferenceManager or the RPC handler. The current regression calls the controller directly and misses this path.
There was a problem hiding this comment.
Fixed in f6c1bec: the weights are downloaded in their own daemon thread, outside the controller's management worker, so other LOAD/UNLOAD operations are not blocked and stop() does not wait for the download; a download that finishes after stop() is dropped. 584f55d closes the UNLOAD ordering gap: UNLOAD MODEL on a model that is still downloading is queued and runs once the load finishes, instead of returning success without effect. Both are covered in test_builtin_model_load.py.
CRZbulabula
left a comment
There was a problem hiding this comment.
Reviewed e3475fa. I independently reproduced the three issues in the existing review threads and added the evidence there: synchronous downloads block the shared controller, failed configuration processing still reports download success, and batching independent series causes quadratic mask-memory growth across requests. These should be addressed before merging.
All ten new Python tests pass locally (six model tests and four download tests), using Python 3.11 / PyTorch 2.11.0 on CPU. This does not validate the pinned PyTorch 2.14.1 or GPU behavior. The current AINode CI job is green, but its explicit Python test commands only select test_dependency_security.py and test_sundial_flow_loss.py; please include both new regression suites in CI as well.
I replied separately in #19270 about the optional download design and the remaining weights-license question.
CRZbulabula
left a comment
There was a problem hiding this comment.
Thanks for the follow-up commits. The malformed-config result and unbounded cross-request batching issues are fixed, and the new test suites now run in CI. All 15 new Python tests and the 19 packaged integration tests pass in the latest AINode job.
Besides the pending-load/unload issue noted in the lifecycle thread, the weights-license question remains open. The restriction is documented in the PR description, but there are still no user-facing setup instructions explaining the eligibility condition before LOAD MODEL downloads the weights. Please add those instructions with a link to the exact checkpoint license. We also need to close the maintainer/PMC assessment of this optional download workflow; an explicit download does not itself resolve the license restriction.
After #19273 lands, please rebase this PR: the two changes conflict in model_info.py and the license/notice files. Both model registrations and the max_output_length / download_on_start fields need to be retained and tested together.


Description
This PR adds Timestral (Mistral AI) to AINode's built-in forecasting models, next to
timer_xl,sundial,chronos2,moirai2,totoandmoment. Timestral is a patch-based transformer for multivariate probabilistic forecasting. It forecasts 21 quantiles (0.01 to 0.99) from a context of up to 32,768 steps, and it can use past-only and future-known covariates. The model is registered astimestral, with the weights ofmistralai/Timestral-209M. The 209M model is the largest of the four published sizes (5M, 28M, 123M, 209M) and the one Timestral's own documentation uses.Unlike the other built-in models, AINode does not download Timestral's weights at startup. They are downloaded when a user loads the model with
LOAD MODEL. The reason is the weights' licence, below.Weights licence
Timestral's code is under the Apache License 2.0. Its weights are not. Every published size is under a "Modified MIT" licence (licence text), whose second condition reads:
The weights are not part of the AINode distribution in either case. However, AINode downloads the weights of every built-in model when it starts. As an automatic built-in, Timestral would put weights that some users are not licensed to use on every AINode installation, without anyone asking for them. With this PR, the weights reach a machine only when someone runs
LOAD MODEL timestral, which is a deliberate act by that user. The same question applies tomoirai2, whose weights (Salesforce/moirai-2.0-R-small) are under CC BY-NC 4.0 and are downloaded at startup today. The newdownload_on_startflag could be applied to it as well. That is left to the maintainers, and this PR does not changemoirai2.The maintainers may prefer a different policy, such as not shipping models with restricted weights at all. The Timestral integration does not depend on the on-request download, so it can follow whichever policy is chosen.
Usage
Once downloaded, the weights stay in AINode's model directory, and the model is active from the next start on.
Example: the input is 1024 points of
50 + 10·sin(2πt/24) + 0.01·t. The request is aTForecastReqwithoutputLength = 24, sent to AINode's inference manager on CPU with the Timestral-209M checkpoint. The response is 24 values (the median forecast). The first six:The mean absolute error is 0.031 over 24 steps and 0.049 over 96 steps.
On-request download of built-in models
ModelInfogets adownload_on_startflag, which isTrueby default. Every existing built-in model keeps it, so their behaviour does not change.ModelStorageregisters a built-in model withdownload_on_start=Falseasinactiveat startup, unless itsconfig.jsonandmodel.safetensorsare already in its directory.ModelStorage.download_builtin_modeldownloads the two files and marks the modelactive. If the download fails, or the downloadedconfig.jsoncannot be read, the model goes back toinactiveand the load stops. The download code is the existing one, moved into a method.LOAD MODELstarts the download in its own thread, and the pool controller starts the inference pools when the download ends. TheLOAD MODELstatement returns immediately, as it does today.SHOW MODELSshowsactivatingduring the download, andSHOW LOADED MODELSshows the model once it is loaded.UNLOAD MODELcounts a load that is still pending as loaded, so an unload sent during the download is queued after the load instead of being skipped. Stopping AINode does not wait for a download either.FORECASTandCALL INFERENCEon such a model before the download return error 1504 with the message shown above. Without this check, they would fail while loading the missing weights.ModelInfogets an optionalweights_license_notice. For Timestral, it links the checkpoint's licence and states its revenue condition. The 1504 message includes it, andLOAD MODELlogs it as a warning before the download starts, so users can check that they may use the weights.Model code
iotdb/ainode/core/model/timestral/contains the inference code from mistralai/timestral 1.0.0 (Apache License 2.0), in the same way aschronos2/contains the code from chronos-forecasting. The model, forecaster, layers, masks, scaler, covariate encoding and configuration are unchanged except for these points:iotdb.ainode.core.model.timestralpackage, and files carry the ASF header followed by Mistral AI's copyright notice.loading.py) andTimestralForecaster.from_pretrainedare left out. AINode downloads and loads the checkpoint itself.AINode integration
modeling_timestral.TimestralForPredictionis the model, loaded throughPyTorchModelHubMixin(hub_mixin_clsinModelInfo), a loading path AINode already supports.config.jsonis read into Timestral's ownTimestralConfig, which rejects unknown keys. The weights load withstrict=True. The rotary-embedding frequencies are recomputed whenever the model is moved to a device, as Timestral's own loader does after moving the model.pipeline_timestral.TimestralPipelineruns Timestral'sTimestralForecasterwith its default settings:past_covariatesbecomes a past covariate. A covariate that also appears infuture_covariatesbecomes a known covariate, built from its past and future values.output_lengththat AINode accepts (up to 2880) works.Dependencies
There are no new dependencies. Timestral needs only PyTorch, NumPy, safetensors and huggingface_hub, which AINode already pins.
pyproject.toml,ainode.specandainode.xmlare unchanged.ainode.specalready collects every module underiotdb.ainode.core.Licences
The code is under the Apache License 2.0 (licence). This PR adds it to
LICENSE,NOTICEand AINode'sNOTICE-binary. See the note above about the licence of the weights.Tests
tests/test_timestral.py(unittest, CPU only, no downloads) uses a two-layer Timestral with random weights. The tests:TimestralPipeline, and check the output shapes for 1-D and 2-D targets over more than one decode block, and that the result is the forecaster's median;tests/test_builtin_model_download.py(unittest, no network:hf_hub_downloadis mocked) checks that:inactiveand not downloaded at startup, while the other built-ins are;download_builtin_modelfetches it and marks itactive, and a second call downloads nothing;activeat the next startup once downloaded;config.jsonthat cannot be read, leaves itinactive;tests/test_builtin_model_load.py(unittest, no downloads) checks that during a download the pool controller still runs other models' loads and unloads. It also checks that the downloading model's unload runs after its load, that a failed download skips only the load, and that the controller stops without waiting for the download. ThroughInferenceManager, it checks thatLOAD MODELfollowed byUNLOAD MODELduring a download unloads the model after it is loaded. It also checks thatLOAD MODELlogs the licence notice for Timestral only.test_dependency_security.pyandtest_sundial_flow_loss.py.AINodeTestUtils.ON_REQUEST_BUILTIN_MODEL_MAPliststimestral. TheSHOW MODELStest accepts it in either state, and the newonRequestBuiltInModelLoadInferenceAndForecastTestloads it on the CPU, which downloads it, then runsCALL INFERENCEandFORECAST.SHOW MODELSfirst reportsinactive, andFORECASTreturns 1504.LOAD MODEL timestral TO DEVICES 'cpu'downloads the files and starts one request pool.FORECASTthen returns the 24 values above through that pool.This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR
iotdb/ainode/core/model/timestral/(new): the Timestral model and forecaster,TimestralForPrediction,TimestralPipelineiotdb/ainode/core/model/model_info.py:ModelInfo.download_on_startandweights_license_notice;timestralentry inBUILTIN_HF_TRANSFORMERS_MODEL_MAPiotdb/ainode/core/model/model_storage.py:download_builtin_model; built-ins withdownload_on_start=Falseare not downloaded at startiotdb/ainode/core/inference/pool_controller.py:LOAD MODELdownloads such a model in its own thread, then loads itiotdb/ainode/core/rpc/handler.py: forecasts on a model that is not downloaded yet are rejected, with the weights licence noticeiotdb/ainode/core/manager/inference_manager.py:UNLOAD MODELis forwarded for a load that is still pendingiotdb-core/ainode/tests/test_timestral.py,tests/test_builtin_model_download.py,tests/test_builtin_model_load.py(new).github/workflows/cluster-it-1c1d1a.yml: runs the new test filesintegration-test/.../ainode/utils/AINodeTestUtils.java,.../it/AINodeSharedClusterIT.java:ON_REQUEST_BUILTIN_MODEL_MAPand its testsLICENSE,NOTICE,iotdb-core/ainode/src/assembly/resources/NOTICE-binary